A VPS is cheap because it is a small compartment of a big dedicated server. The resources of the parent server that are leased are shared among webmasters. But, you get fixed amount of resources that are allocated only to you. A low cost vps can be a good starting point if you are starting off with your website.